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Final Abstract 

^ -* « 

The International Reading Association (IRA) revised the Stuffl&beam 
< joint committee's "Standards for Evaluations" to assist the reading 
profession in interpreting reading program evaluations.' Under , 
' development by IRA for a year prior to field test, the adaptation 
work resulted in a draft document. This project involved a field 
test -^of the IRA adaptation in four local school districts. The 
purposes of the f|eld test were to assess the usefulness q£ the 
adapt'ed standards : and to enable IRA further to improve and ta v ilor 
its work tp local school 'personnel 's needs. 

School districts currently conducting or interpreting an evaluation 
of their local reading program participated. Local staff uere - 
oriented to the IRA draft "Guidelines" and then applied them in 
their districts. Project staff gathered feedback concerning the 
Guidelines from the local .districts during and immediately after 
the^ field test.. Based Oh the results, the Guidelines wer>£ revised, 
reviewed by various IRA Committee members, and prepared in final 
draft form. They will soon be published by. IRA and distributed to 
Other disseminaxic 



its membership. 



Ion procedures are underway. 



Local ^school personnel consider reading program evaluation to be 
an im&ortant activity. However, most districts are without access 
to professional assistance in interpreting evaluations. Eased on 
the many requests IRA re<feives for assistance of this type, the 
publication that results ffrom- this project has the potential for. 
guiding a^ large .number of iQJfal districts toward better use of 
^evaluations . The field study enabled IRA to conduct am empirically 
based revision cycle for the Guidelines, thus making them more 
useful to local school district^; 



/ Executive Summary 

vAn Adaptation of the "Standards <for Evaluations 11 for the Interpretation^ 
of Reading Program Evaluations (NIE-G-80-0075) * * f • 
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I. PROJECT BACKGROUND , , ' ' . . 

Evaluations, of educational programs and materials are an impdftant concern of 
almost all school administrators. Although many procedures for conducting such 
-evaluations Have evolved, most local school pe^onitel are not trained in these 
techniques. Further, local staff is often even less equipped to evaluate an 
evaluation to analyze the procedures used, data included, and conclusions , 
reached. It is toward this latter problem 7 - interpreting an evaluation report 
^ that this proposal is directed. ■ v 

In 1978, the, International Reading Association (IRA) organized a Committee on the 
Study of Reading Program Evaluations. Thi$ Committee, had as. its primary -charge tfo 
develop "criteria to be used to judge the adequacy of the evidence to support the 
superiority of a, particular reading program... M .The Committee used the Standards 
for Evaluations of Educational Programs; Projects; and Materials document as >a 
starting point in evolving ta written set. of "Guidelines for Reviewing Evaluation * 
Reports" to be used by people attempting to assess an evaluation report. This 
work went through three ■ revisions prior to the/ field test funded by this grant . 
The purpose of the field tyial was to assess fc the Guidelines 1 usefulness $nd improve 
upon them before making available a revised set of Guide Lines to tRA's general 
membership. • , 

IRA members are involved in the interpretation and use gf literally thousands of 
reading program evaluations annually. The adaptations we propose to publish, \ 
assisted by this funding/ will find wide use among our 70,000 members.' further; 
this work will improve both the understanding of evaluations by school personnel' 
and the interpretation and application of such evaluations on a local level.- 

II. PROJECT ACTIVITIES* * * , . 

The IRA* Study -of Reading Program Evaluations Committee conducted this project, 
with assistance from the* IRA Research Department. There were essentially eight 
stages to the project, as outlined below. 



the Guidelines .dfaft and Field Test plans . t this was completed in 
'An^ijit^rim set of Guidelines was developed for review in May, 1980. 



1 . Finaliz e 
two stages, 

Th^se Guidelines were then further revised and adapted at an August, 19-5TT meeting 
of the, Committee. *Also^ at that time, general plans for the .field test were discused, 
*site* selection procedures and materials were finalized, and plans fot the orientation 
for local field test oo*ordinators, were made. 

2. Site Selection . During .September , 1980,, 105 IRA local council pjresisients wore . 
sent information concerning the- field test , soliciting the participation of their 
member school districts. A second mailing was made to a somewhat-expanded list of 
local cgancils during October. The*se*two mailings resulted in approximately 20 
inquiries or expressions of interest. From this number, a list of five districts 



from four stated with varied evaluation experiences/ materials, and interest was 
♦determined. These districts *were formally invited^ to participate. Since one of 
these districts later was forced to withdraw,* the field test was conducted with 
four sites.* ' * 

" . " • ' ' : / 

-3; fomentation for Locgl Districts . This meeting was held January 18-20, 1981 » 
at IRA headquarters in Mewark, Delaware. At that time^, committee members aftd^ 
appropriate IRA % staff members met; vfith representatives* of the four %ield~ test *♦ 
districts. These districts were: Webster (NY) Central Schoo^ District ,' Rock 
Falls fwp. (IL) High School District, Houghton Lake (MI) Community Schools, and 
Downers Grove (IL) Grade School District." 

During this meeting, 'the project was explained to the field *tes^ coordinators, 
the draft Guidelines were discussed in detail, and lotal implementation procedures 
for the field test were discussed. One committee member was assigned -to .each 
locSl district as its "IRA contact person/' Based on feedback received at this 
meeting, ' additional revisions were mad e^ .to the Guidelines. The Guidelines revised" 
as a /result^ of^the orientation meet ingTwas 'the $et used for the field test ; 

4. Field Test . Following the meeting in Delaware; each of the field .test 
coordinators conducted informal "orientation sessions" •with several persons in 
their, district who had interest iji Qr resjfonsibi lity for some, aspect of a reading' 
program ^Valuation. Then various local personnel were asked to review independent! 
the Guidelines draf£ document. The focus of the review was dn how the Guidelines 
could best be improved in content, organization, scope, or wording. The — local 
reviewers were 'also asked to' complete a "Comment-s % Reactions," page. 

The review led to several conclusions concerning tHe Guidel-ines"*draf t . Rirst , 
selected'portions of the Guidelines required additional clarification, edrting,' * 
and/or rewriting.' Second, despite these many suggestions, the "reviewers generally 
found the document to be very comprehensive and clearly organized, readable and 
reasonable for school use/ and sufficiently non-technicai . The primary negative 
reaction received from reviewers was that the Guidelines were almost too compre- 
hensi\re and needed to 'be shortened somewhat , ^with jargon eliminated." # 

5. Site Visitations . Two site visits were scheduled for sach field test district . 
The first wa^s to be made by the Comtnit*tee urember a'ssighed' to • the district the 
second -vi,sit was to -be 'made by . a Rese^.rch|Specialist from.JRA headquarters. The' 
purpose of the two visits was the, sauje: 'to meet with/interview Ideal p§gpie who 
had participated in the field test review process to obtain their reactions tq and 
suggestions for the Guidelines'. . • / \" V ' 

Both Committee members and local Coordinators reacted well to the format and scope 
( of the visits and no difficulties were encountered. Further, local reviewers gave 
willingly of their time for these interviews and wiced . positive reactions both to 
the project 'or their district's participation. > § 

. * , ■ * 

6. Committee Revision of the Guidelines . During the IRA Jihnual Meeting, Committee 
members met to revise the Guidelines. Be*foxe this^tvne, Coiftmittee members had sub- 
mitted written summaries of their site visits for teview. Both genernJL topics ^ 
concerning the purpose ,' audience, Overall tone, and seopg of tj^e guidelines , 4 as wel 
-as specific wording and rewritti'ng suggestions were explored, Guidelines draft's* 
reviewed by the various local personnel were- used, to make revisions in the. draft. 
Many changes made in the Guideline? at this stage, based on the field test results.. 



7 '• IRA Revie w of the Revised Guidelines . A revised version of the Guidelines 
draft was sent to. WJ selected IRA /volunteers, ' including chairpersons pf each 
.of ; #ie 48' I,RA Standing-. and Ad Hoc Committees and all 'members of s four committees, 

, the memberships of which is -heavily" weighted with pe.ople interested" in .and knowledge-, 
able about evaluation... The 101 people were asked to "critically review' 7 the draft 
and make either editorial or substantive comments. They were also asked to complete 

, a "Comments § Reactions" form similar to that used in the field test. • " 

8. Final Draft . Although this phase of the project was -not' covered bv ME funding,, 
nor does it fall within the activities of the grant, it is outlined here for closure 
purposes. As with the field test responses, the most worthwhile .portions of the IRA 
review were the comments item ,b'y item and major section by .section. These comments 
were used to pull together yet another revision of the Guidelines which is currently 
being reviewed by Committee members. The proposed .final draft "will be submitted to 
the IRA Board of Directors! in March 'for approval. This "final draft" version will 
■be referred to the iTtA Publications Committee for dissemination. 

r •* •• ' ' 

UII. ANTICIPATED USES A.ND USEFULNESS OF THE GUIDELINES .' 

% In general, ye would make the following observations concerning the anticipated 
usefulness of the. final: Guidelines: * ' *~ 

1. The Guidelines as a whole are K>o comprehensive for most local evaluations. 
Maximum usefulness will probably result 'if a* small group of Jocai personnel 
select the relevant portions of the Guidelines and the entire set is reduced to' 
some limited subset of 'items especially pertinent locally. This .is no* suggested ■ 
in the Introduction. - ■ 

2. The two primary uses of the Guidelines will probably be: fA) in reviewing plans 
for an internal evaluation before the evaluation begins'-- looking for additional 
componentsr to be considered; and fB) in judging the quality of an external evaluation 
after it has been completed. We believe that, for these two purposes, th$ -Guidelines 
will be helpful . as a stand-alone document for most districts. 

3. Use of the Guidelines for either the planning or evaluating function would \ 
w probably be improved if an orientation to the topic and document was. a?v-ailable. 

We will continue to explore this impression through^the sessions to be conducted 
over the next several months using th^Guidelines . 

IV. PROJECT FUNDING AND STAFFING 

The project was funded for the budgeted total of 511,082. Despite greatly increased 
travel costs during, the* contract period, the total cost to N^ 'of this project was 
$9S51,64. All professional time for this project* except that spent by our IRA 
headquarters Assistant, was donated to IRA. Records of this % ti£e were' kept by each'- 
Committee member and 1 local field ^est coordinator during the project. These items 
totaled to 137 professional days. It is clear .that the. project was completed^l 1 ^ 
minimal cost due largely to the donation of time by these concerned professionals, 
all of whom we sincerelv thank. 
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.Figuring donated staff time at an -estimated S130 per day, including benefits, and 
- including no additional overhead expenses, it is estimated that the total "cost" 

ofy this -project was oyer 531,000, only 30% of which was charged back to N IE . 
4 Projects' such as this, seem to be an ideal way for governmental agencies to derive 
maximum impact for their shrinking contract/grant funds. 

DISSEMINATION AND DISTRIBUTION PLAN 

Clearly, the ultimate ^impact of our efforts will depend on the extent to which 
the Guidelines are disseminated to the IRA membership and interested others. 
We have been keenly aware of this throughout the project and have already finalized 
the following plans: 

* 

A - " Presentations at Conferences . -Sessions have .been* proposed and accepted for 
presentation as, of this date at three conferences. Presenters will be either 
Committee members or participants in the field test. In addition to these 
currently scheduled sessions, two proposals for, additional presentations are 
planned. Feedback from these sessions will assist IRA in additional refinement of 
the Guidelines for publication. 

. . ' , ' / 
B. Publications . The final draft of the Guidelines will be. submitted by the 
Committee for approval by the. IJIA. Board of Directors;. Following approval ,' the 
Publications Committee will decide , on- the means of distribution for the Guide- 
lines via the IRA publications system. Finally, an article describing the develop- 
ment of the Guidelines will be submitted to The Reading Teacher »or Journal of ' 
Reading for publication. All of these efforts Should ensure the wide distribution 
of the Guidelines to 'interested professionals. - 
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An Adaptation of the ^Standards for Evaluations" for the Interpretation 
of ^Reading Program Evaluations (NIE-G-80-0075) * 

c ■ ' 

I. PROJECT BACKGROUND ' ' ' - 4 / ; 

t 

Evaluations of educational programs and materials are an- important concern oF 
almost all school administrators. Although, many procedures for, conducting' such 
evaluations' have evolved over . the past 30 years or so, most local school personnel, 
.are not trained in these techniques. Further, local staff is often even less 
equipped to evaluate an evaluation to analyze the procedures used, data included, 
^nd conclusions reached. TFls^owaaedJhis latter problem — interpreting an 
evaluajtitfrf report that this proposal is^Ji^reetr^T ■ > 

- ' s ' 

The recently completed, Standards for -Evaluations of Educational Programs, Projects , 
and Materials (McGraw-Hill, 1931) were developed to fill the widely perceived need 
for standards for program evaluation similar to the APA-AERA-NQE standards for 
published tests. These Standard's for Evaluations, partially funded'by the National 
Institute of Education, may eventually find wide_ use in guiding the conduct and 
evaluation of educational evaluations. However^ the Standards in their complete 
for^ are very comprehensive , and broadly based. 

Because of this, "and because of frequent requests from members for sluch assistance, 
in 1978 the International Reading Association (IRA) organized' a Committee on the 
Study of Reading Program Evaluations.* This Committee has had as its primary * 
charge to develop "criteria to be used to judge the adequacy of the evidence to 
support the superiority of a particular reading program*.," The Committee usecf 
the Standards for Evaluations document and various othet evaluation guidelines as 
starting points in evolving a written spt of questions, termed "Guidelines for 
Reviewing Evaluation Reports," to be used by people attempting 'to assess an 
evaluation report*.. This work went through three revisions by the Committee . prior 
to the field^test funded ,by this grapt. The purpose of the field trial was to N 
assess* the Guidelines 1 usefulness <and ^ improve upon them before making ayai lab 
a revised set of Guidelines to IRA/s general membership. 

IRA members are involved , in the interpretation and use of literally thousands of / 
reading program evaluations annually. It is our belief that «hfe adaptations we 
propose to publish, assisted by this funding, will find widf use among our 70,000 
members. Further, this work will serve to improve both the understanding of 
evaluations by schcJol personnel and the interpretation and application of such 
evaluations on a local 1'eVel. . 

Obviously, there is a multitude of types of evaluations available to school 
personnel. Such variables as whether the evaluation is internally or externally 
conducted; its duration and .scope; whether it contains "qualitative, quantitative-, 
or bgth*- types of data; whether its purpose is to assess ongoing programs or ' / 
propose revisions; etc* all distinguish various types of stich evaluations. The ' 
Guidelines developed clearly cannot be equally valuable in interpreting all- types' 
of evaluations. However, e they 4£e generally Applicable to internal and external, * 
short and long-term, and quantitative and qualitative formal, written evaluations. 
We believe that they will assist local personnel in assessing the content, of most . v 
oflthe^wide variety of evaluations with which they come in frequent contact * * . 



II. PROJECT ACTIVITIES ■ f ■ \ 

The International Reading Association (IRA) Study { of. Reading Program Evaluations 
Committee conducted this project, with ^assistanqe frbm the IRA Research Depart- 
ment. There were essentially eight stages to the, project > as outlined below. 1 

Finalize the Guidelines draft and Field Te£t plans . This was .completed in 
two stages. An interim set of Guidelines was developed for review in May, .1980 
by Committee members. These Guidelines were then further revised and adapted 
at an 'August, 1980 meeting of the Committee in Indianapolis. AIsq at tKat^time, 
general plans for the f.ield t^st werp discussed, site selection procedures and 
materials were finalized-,- and plans for th« orientation for local field test 
coordinators were padre. This activity was completed essentially as originally 
planned and scheduled. NIE funding financed the Committee's August meeting. 
Committee members and their current professional affiliations are' listed below: * 

- Michael Becl^, Chair Alden Moe, IRA Board of 

Senior Project Director Directors Liaison \ 

The Psychological Corp. * Professor of Education 

Purdue University 
Mary Se if ert,- IRA Headquarters •> 
Research Specialist Rebecca Barr ' ' 

International Reading Association Department of Education 

University of Chicago 
Roger Fa*r * . ^ • * 

Director^ Smith Research "Center * . Jerome Harste 

* Indiana University ■ . Assoc. Professor of Reading 

Education 

Frank Ferris ' Indiana University 

Director of Reading 
t . Spencerport (NY) Central Schools > Susan Stuber 

Title I Director 1 4 

* Edward Smith . Oscoda (MI) Area Schools 
Basic Skills Improvement Pffice-DOE 

Washington ,* D.C. t I 

. * <* 

2. Site -Selection . During September", 1980, 105 IRA local council presidents 
were sent, information concerning the field test, soliciting the participation of 
their member school districts, <We had anticipated a faster and larger return 
from this mailing than we' actually experienced. For both of these reasons, a 
"second foljow-up mailing was made to a somewhat -expanded list of local councils 
during October. These two mailings resulted in approximately 20 inquiries or 
expressions of interest. From this number, seven apparent good candidates were 
selected and Contacted for more information concerning their evaluations work. 
Finally, a Hst of five districts from four states' with varied evaluation 
experiences," materials, and interest was determined. These^districts were 
f6rmall r y invited to participate. .Unfortunately, one of the^e districts later 
was forced to withdraw due to a new local board of education ruling prohibiting 
travel outside the district by any staft member^regardless of cost reimbursement* 
Despite discussions with the superintendent on tj^ matter, we were unable to 
obtain exemption for the project. Due to the ljj|ness of the date ana the lack 
of ^n appropriate substitute district, the field^test was conducted with four 
sites. * 



3 - Orientation for Local Districts . The schedule for this meeting .was finalized 
when- the site selection activities were completed. The meeting was held January 
18-20, 1981 at IRA headquarters in Newark, pelaware.' At that time, committee ' ' 
members and appropriate IRA staff members met with representatives of the four 
field test districts. These districts and the local field test coordinators .. 



were 



Webster (NY) Central School District - Janice Strege w ■ 

~ Reading Coordinator 

Rock Falls Township (IL) High School -.Edward Mulvaney 

* - . \ "Reading Specialist 

Houghton' lake" (MI) Community Schools - Ken Roberts 

Ctirriculum/Federal Projects Dir. 



Dd^ers Grove (IL) Grade School Dist - Ronald 'Hfcle 

, Asst. Supt. for Instruction 



Curing this meeting, the project 'was explained to the field test coordinators, 
the draft Guidelines were discussed in detail, a' Sample' evaluation report* was 
"rated" using the Guidelines} and local implementation procedures for the field 
test^ were discussed. One IRA committee number was assigned to each local district 
as. its "IRA contact person." This person was in charge of maintaining contact 
with the local field test coordinator toncerhing the project and with making a 
site visit to the district. Based on feedback received at this meeting, additional 
revisions were made to the Guidelines: Thd Guidelines revised as a result of the 
orientation meeting was the set use.d for i*ie field test'. Although we had not 
originally planned to make arevision at this point, the concensus* of opinion was 
that the revisions were important and that the usefulness of the 'field test would 
be increased if the revised version was used. A copy of the 'field test Guidelines 
is presented in Appendix A. ; 

The meeting wa$ considered very successful by Committee members and field test 
coordinators, and we bejieve it improved significantly the implementation of the 
field test and the results obtained therefrom. This meeting was funded by the 
NIE grant. Following the meeting, the IRA public information office sent a press 
release describing^the field test to the hometown newspaper of each Committee 
member ami local- field test coordinator These stories, which mentioned NIE'S 
support for the project, highlighted the cooperation of the local districts in the 
field test. 

4. Field Test . Following the meeting in Delaware, each of the four field test 
coordinators conducted informal "orientation sessions" concerning the -field test 
with several persons in their district who had interest 4n or responsibility for' 
some aspect of a reading program evaluation. The^purpose for the development of 
tjte. Guidelines was discussed, as well as the intent that* the Guidelines be 

/generic'," rather than being tailored to any particular type of evaluation. 
, Following the orientation, all local personnel were as'ked to review, independently 

( the Guidelines draft document. The focus of the review was on how the 

| Guidelines could best be improved in content, organization, scope, or wording. 
The reviewers were mad* aware that both fine, "editorial" suggestions as well as 

^ more global, ^philosophical" comments were being sought. -Reviewers were asked to 
make their comments with their own local evaluation concerns in firind, while 
remembering that the final Guidelines were to be used with various types of 
evaluations. After working through the Guidelines item by item, local reviewers 
were asked to complete a "Comments § Reactions" page. 



A total of 22 local personnel (4-7 per field test district) completed a review i 
of the Guidelines; of these, 19 people completed the "Comments, and Reactions- 
form. According to primary job responsibility, the 22 local reviewers included 
6 classroom teachers, 4 principals, 3 reading teachers, 2 reading! coordinators , 
2 curriculum directors, 2 counselors, and 1 each superintendent, assistant 
superintendent,, and test director. Following the review, the lbcaP'field test ' 
coordinators collected the Guidelines and reactions 'forms and returnedahem to 
their IRA Committee contact person. A summary of the "Comments and Reactions" ' 
form is presented in Appendix B. . 

The review resulted in the Committee reaching several ' conclusions concerning the 
Guidelines draft. First, selected portions of the .Guidelines required additional 
clarification, editing, and/or rewriting.' Second, despite these many e'ditorial- 
type suggestions, the reviewers generally found the document to be very compre- 
hensive and clearly -organized, at least somewhat readable and reasonable for 
school use, and sufficiently non-technical. Many reviewers surprisingly indi- 
cated that the document woufd be more (or at least as) useful in planning an 
evaluation as in interpreting an evaluation already completed. This led to 
much discussion in }ater committee work. The primary negative reaction* received 
from reviewers was that the Guidelines were almost too comprehensive and needed 
to be shortened somewhat, with jargon eliminated. * ; 



f 9 i. 

5 ; Site Visitations- . Two site visits were scheduled for<each field test 
district. The first was to be -made by the- Committee member assigned to the 
district; the second visit was to be made by Mary Seifert," the Research Specialist 
from IRA headquarters. The purpose of the. two visits was. the. same: to meet with/* 
interview local peoplfe who had participated in the field test review process to 
obtain their reactions to and suggestions for the .Guidelines . . Two visits were 
used since we considered it' likely that somewhat different qualitative 'evaluations 
would be obtained if two different people met with district staff at two different 
times. '• 

- " f 
The "formats" of the site visits varied between, and often within, the districts. 
That is, for some site visits, tfie IRA Committee representative met individually 
with the local reviewers. In other instances, a group meeting was used in which 
the local field test coordinator participated. Still other" visits were group 
meetings excluding the local cArdinator. Arranging the formats of the site visits 
was the responsibility of thoTlocal coordinator, although, they were urged to vary 
the format for the two visits. t 'Both Committee members and local . coordinators 
reacted well to the format and scope of the Visits and no difficulties were 
encountered. Further, local reviewers gave willingly of their time 'for these 
meetings or interviews and in mo'st instances voiced positive reactions both to 
the project or their district's participation. " * 

r , * 

Due to various schedule conflicts and the variability of l»cal progress on the 
review of the Guidelines,- one of the four field test sites was visited only once. 
The Principal Investigator interviewed" the local field test 'coordinator via phone 
in this instance as the "Second visit." Although the failure to make a- second 
visit at this-site was unfortunate, it is not likely to have affected materially 
the results of the field test. " ■ . 
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6* Committee Revision of the Guidelines . During the>IRA Annual Meeting, Committee 
members and their Headquarters .Research Specialist met on two' occasions to revise 
the > Guidelines/ Before tjiis time, Committee members and Mary Seifert had submitted 
writtep summaries of their site visits for review. Both general topics concerning 
the purpose, audience, .overall tone,, and scoptf of the Guidelines, as w,eH as 
specific wording and rewritting suggestions were explored. Guidelines drafts 
reviewed by the various local 'personnel were used to make .revisions in the -draft. 
Major changes made iri the 1 Guidelines at this stage, based on the field test 
results were: 

i - » 

**. ' v * 

A. The Guidelines were reduced in length by almost a quarter." The number of 
specific items included in the Guidelines was reduced from 1S6 .to 11$ • Most of 
these eliminations wer£ made because of specific recommendations of the field te'st 
reviewers, as well as the frequent field 'test comment that the Guidelines were - 
, currently too long and complex. * 



B, The field test -Guidelines contained^ essentially two sections for each included 
item — olje indicating ^the ; topic was addressed in planning the report / the second' 
indicating the topic was included in the actual report itself. These dual categor 
were used because of suggestions made ^t the field test jorientation meeting. How- 
ever, the two sets of ratings were considered by many field test reviewers to be 
confusing and/or unnecessary. Therefore, in the revised draft, we returned to a 
single set of categories* - , ,> \ 

C. The tield^test Guidelines topics were worded as questions rather than ,state-- 
ments. This^ issue had been .dispussed at every jucture in the development of the 
Guidelines. Our best judgement at the time of field test was that questions were 
preferable. .However, the field test results indicated otherwise; . and we returned 
to declarative statements, * ^ * 



les 



D. Various changes were made in the wording of well over half of the included 

items. Many of these changes had. the effect of 'deducing the- length and 'the 

I "jargon load" o'f the 'items, -as recommended during the field .test. 
* « » 

E. One of the six sections, "Local Conditions," was reduced substantially in 
length and eliminated as a separate section. Remaining topics were included in 
tlie "Focus 1 * .section. The "Local Conditions"^ section was the most heavily 
criticized portion of the field test draft. 

F. The "Information Sources" section was reduced substantially in, 'length (by 
about 1/3). Many field tejst reviewers had commented that the draft version was 
tog_t£chnical and was unreaftistically beyond the scope of typical school personnel. 

» * 

G. A section was added'to the Introduction t6 more clearly indicate that many of 
the topics included would not be appropriate for every particular local ♦evaluation. 
This is, thq Guidelines, are, intended to be more comprehensive than any given . 
report. Therefore, the best use of the Guidelines^ in' a local situation may be-to 
narrow ^down the entire list of topics to those on which there is some consensus 

as to importance to the local evaluation. 

• # - / / 

Along the same lines, a sentence wa$ added stating that responses of "No" did not 
necessarily imply* weaknesses in the evaluation. Some field test reviewdrs feared 
people would merely tally, tl^e "Yes" and "No" responses and attempt to derive 
some quantitative "score" for the report. Th^s was to be 'discouraged. 1 

<* « 



er|c 



-5- 



11 



'7. IRA Review of the Revised.^Guidelihes . ' Appendix C 'contains a copy of . the , [ 
-Guidelines as revised based^/of} the field test. This version was sent to 101 
selected'iRA voluntetets, "including chairpersons of each of the 48 IRA. Standing 
and Ad Hot: ^Committees and \11 member's v of the following committees: Evaluation of 
Test's , Teacher Education, ^tTjdies $nd Research^ and Interpretation and #^lysis. * 
'Of NAEP Reading Assessments Data*.* These four committees were selected because M 
their memberships are heavily weighted with people t interested in and knofoledgfeable 
about evaluation. ^ * ; 

The,, 101 people were asked to "critically review" the draft and "make either ' 
editorial or substantive comment^ (see Appendix D) . . They were also asked to' 
complete a "Comments and Reactions;"' form syniiar-to that used in the field test. * 
By the deadline for. responding, 4$ t^lies* had been received, including 41 wha 
completed the evaluation form. These' data are * summarized' in Appendix E. 

Concurrently with this review, members of the Committee for the Study of Reading 
Program Evaluations made additional comments 6n this host * current draft of the 
Guidelines. These c'omments were pooled with th*se of the other reviewers. • 

Interestingly and probably not surprisirigly, ^there were several parallels 
between the ratings during "and after the field test. Both sets -of respondent^ 
saw the Guidelines" drafts as being well-organized^ comprehensive, and 'useful as 
an "ideal,." There appears to be significant improvement in -.the'.post-field' test 
Guidelines in terms of the readability of the questions and in the Information ^ 
Sources section, two areas that received much ' comment during the field test. 
(The alternate explanation is,^crf course, that tjjie' IRA 4 respondents ,were a more 
sophisticated group, for whom these areas were less likely to be confusing than 
they' were' to' typical school personnel*. Which, if either, of these expl* 
is closest to the > truth is difficult tp determine.) 



Lar^^i 



ons 



\ 



8. Finaj. ^Draft . Although this phase of the project, is not covered by *NIE funding, 
•nor .does it fall within the activities of^the grant, it is outlined here" for 
closure purposes. As with the- field test responses, ths most worthwhile portions* 
of the IRA review were the- comments item by item and major section by section. 
These copnents were recorded on large summary Sheets End used to pull together 
^et- anpther revision of the Guidelines. This " (hopeful ly) next-to-final draft is 
currently being reviewed and polished by, Committee members art*! the proposed final 
draft will be 'submitted to the IRA Board o'f Directors in March, for approval. * 
This "final draft" version is the one that" will be referred to the IRA Publications 
Commi^tee^ for dissemination $s indicated in the Dissemination and Distribution 
Plan below. ' . 



II L 



ANTICIPATED USES AND USEFULNESS OF THE GUIDELINES 



It may be instructive 'to convey some qualitative perceptions 'gained during the 
field study concerning the anticipated usefulness of the Guidelines in the 
schoolV, vThese perceptions were obtained from the Comments and Reactions forms 
or, more" often, the interviews conducted on £ite by Committee, niembers or Mary 
Seifert. Appendix F contains a summary of*the comments made by field test respon- 
dents to the Cpmments gtnd Reactions question, "As a result of^ising these Guides 
lines, what specific changes, if any, woul4 you* make in any future evaluation 
activities in whic& you might participate?" This provides some indirect indications 



£ the reactions of oui^ field reviewers as to the Guidelines 1 .usefulness . * 
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Additional impressionistic views .concerning the .ultimate utility of Y the Guideline 
were obtained during the fieltf -test site visits. • Selected reactions relevant to 
this question are listed in Appendix G,* . , - , 

Jn'general, vfe w6uld ;make the following observations concerning the anticipated 
usefulness of the final 'Guidelines: ^* . , \ ' 

1» The Guidelines a^ a who fe^ are too comprehensive' for most^local evaluations.' , 
Maximum usefulness will probably result if a small, group of loc^l personnel 
select the relevant portions of the Guidelines and the entire $et is reduced to N 
some limited subset* of items t especially pertinent locally. This is now suggested 
in the Introduction. Most local', noiv-evaluation people would probably find the 
complete, set overwhelming and, therefore, not especially helpful . 

2. Jhe two* primary ^uses of the Guidelines will prqbably be: 

A. # in reviewing plans^for-an, internal evaluation before the. evaluation • ^ 
, % ' * begins looking for additiohal components to be considered. ^ 

B. in judging the quality of an external Evaluation after At has been 
completed / 

We believe that, for these two purposes, the Guidelines will be helpful- as £ 
stand-alone^document . for most districts. 

* * . ^ i 

3.,; .Use of £he Guidelines for either the planning or evaluating function would 
probably be improved if an orientation tortile .topic and document was available, 
We' will continue te explore this impression through th6 sessions we will be * 
'conducting over the next several months using the Guidelines (see Section V). 

IV. ' PROJEOT FUNDING AND STAFFING , 

The project was funded for. the budgeted total of $llv,082. Despite greatly 
Increased trav'el costs during the contract 'Period, the total cost to V NIE of, 
'this project was $9551.64.: Charged t to NIE, compared with budget, werp as 
follows: ^ - S 

* Actual 

Budget- Expenditures 

Personnel salaries/benefits, incl, 

indirect costs > $2,235 $1,640.02 



TraV.el " 8,347 , 1 7,682.99 

Communications ? 500 228 . 63 

7 

• $11,0.^2 $9,551.64 



All professional time for this project, with the exception of .that spent by'our 
IRA Headquarters Assistant, was donated to IRA. ^ Records of this time were kept 
by each Committee' member ( and local figld test coordinator during the project. 
.These times tptaled to 137 professional days- It is clrear that the project was 
completed at a minimal cost to, NIE due largely to the donation of time by these 
concerned professionals, all of whbm we, sincerely thank. 



In ad4ition to the donated .time, an estimated &1600 of actual costs were, paid by 
IRA and employers of th^Cdmmittee members and field ;test districts. These 
expenses included multllith artd other copying charges/ telephone and postage, . 
.clerical and typing support, and miscellaneous ^applies . ' 

Figuring donated staff time at an estimate* $-150 per day, including benefits, and, 
intf^ding no additional overhead expenses; it ig estimate^ that the total "cost" 
of this project was over $31,000 r only 30% of which was v charged back to NIE. 

t Project^sych as this seem to be an^deal way for governmental agencies to derive 

"maximum impact for their shrinking contrlct/grant funds. 



V. DISSEMINATION Alto 'DISTRIBUTION PLAN 



Clearly, the ultimate impact of our efforts win- be determined *by the extent to 
which the Gjiidelircp are disseminated to the IR^membershipttmd interested others. 
We have been keenly aware of this, throughout oiir/wopk on. ^he project, and have 
'already finalized the following plans: J 



A* Presentations^^C^gonf erences . Sessions have *been proposed and accepted, for 
presentation as of this date at three conferences. Presenters will be either / 
^Committee numbers or participants in the fi^ld test. The currently scheduled 
presentations are; 

- _ MEETING > DATE PRESENTER (S) 

^jjjf Illinois Reading Council • March 18-20, 1982 Ed Mulvaney . 

• " ^fhig* 1 Compensatory Education August 19-20, 1982- Roger Farr, Susan Stuher 
4 directors Workshop \ \. v ' 

* ^New York State Reading Conference November 3-6,' 1982 - Mike Beck, Jan St rage, 

b : %' Mary Seifert 

All of these presentations will be workshop-type sessions in which the development 

of the Guidelines- will be discussed, followed .by' a handsr-on discussion of the 

Guidelines 'with participants. Feedback from these sessions will assist IRA in 
< additional refinement of the Guidelines for publication.- 

In addition to these currently Scheduled sessions^ two proposals for additional 
presentations aje planned. These presentations, Mil be at the IRA national 
convention in Anaheim in May, 1983 and at the IRA Great. Lakes regional convention 
✓in 'October, 1983. Attendance at these .two conventions will exceed 20,000, thus 
providing an excellent ■ forum for the dissemination of the Guidelines. These 
1983 presentations will be made using the final version of the Guidelines which, 
hopefully, will have been published by IRA by that tiifie (see b^loV}. NIE's 
support of the field test activities will be acknowledged in each professional 
presentation made* * 

- # 
B. Publications . The final draft of the Guidelines will be submitted by the 
Committee for approval at the April Board of Directors meeting. We wjj.1 request 
at that time through the Committee's Board Liaison, Alden Moe, that the 
-Guidelines be referred to the Publications Committee for printing and distribution. 
Our intent is that tfre Guidelines be distributed in booklet form as a regular 



-publication of IRA. For the several booklets of a similar nature already 
puhlished by IRA, over 350,000 copies have been distributed. These booklets are 
regularly promoted in various IRA publications received by the Association's 
50,000 individual and 20,000 institutional members. 

<-s ' * * 

Following approval, the Publications Committer will 'decide on the means of distri- 
bution for the Guidelines via the IRA publications system. . IRA will send a notice 
of the availability , of, the Guidelines to each of IRA's State Coordinators, asking 

•that the notice be published or distributed through whatever channels are appro- 
priate to the several hundred Local Councils. Finally, an article describing the 
development of the Guidelines will be submitted to The Reading. Te acher or Journal 
of Reading , ^RA journals, for publication. This article will also contribuTe~to~~ 
the dissemination- of the work to interested professionals. All of these 'efforts 
shou Id-ensure the wide distribution of the Guidelines .to HRA members.- 

Copies of the Guidelines -in their final form wii; be submitted- to ERIC and to ME ' 
for possible distribution through the National Diffusion Network. IRA will also 
announce the availability of the materials to a wide variety of professional 
educational associations with which 'it maintains regular contacts. 'Th^se groups 
wiil_ include- curriculum organizations such as the National Council of Teachers of 
English and the Association for Supervision and "Curriculum Development;- research 
organizations such as the .American Educational Research Association, National 
Council on Measurement in Education,- the" Evaluation Network; and other education 
agencies suclyas the Council for Exceptional Children, .American Federation of - 
Teachers, National' Education Association, and the American Association of School 
Administrators. Finally the Guidelines will be shared by IRA with fellow members 
ot the National Consortium on Testing, a national group of over 40 professional 
organizations concerned with test-related issues. All of these activities will 
broaden greatly the potential impact of the Guidelines on the genefal education 
and research community. 
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* APPENDIX A. Draft of Guidelines Used for the Fi-eld'^eSt January 1981 

INTERNATIONAL READING ASSOCIATION, 800 Bark sdale Road POBo* 8139 ti'ewark Delaware 197 1 1 USA x 
ASSOCIATION INTERNATIONALE PQUR LA.LECTURE ^ % Telephone 302 731 I600f 

ASOCIACION INTERNACIONAL OE LECTURA ^ Cable Reading Newark Delaware f 

) t COMMITTEE CORRESPONDENCE 

STUDY OF REAI^ING PROGRAM EVALUATIONS COMMITTEE 
* * Guidelines Fiel& Test. 

The purppse of the attached Guidelines document is to- assist local 

school district personnel in planning, -col!8uc£ing, and interpreting 

an evaluation of a reading program. The Guidelines are designed 

primarily to provide. a frame of reference for assessing the content - 

and scope of sueh an evaluation. ^ 

i - ' . A 

■ Th* format of the Guidelines is a set gf questions*, followed by 
boxes in which you would, respond "Yes" or "No" o*|"Not Applicable," 

1 The .boxes are further divided into Planning luid Import sections. 
A given procedure might have been considered or 4 dteumented in the 

.planning of an" evaluation, but not included 'ig tKe final repor.t. . * 
In such a case,' you would check "Yes" for* Planning and "No" under 
Report. Obviously, pther combinations of responses are possible. 
Questions thai do not seem applicable to .your Jocal evaluation 
efforts should be so checked. Finally, you need not necessarily 
feel constrained by the Yes-tto format wW provide. Other responses * 
or 'annotations^ that may be more appropriate for you should be used 
, if desired. ✓ K \ 

4 It should be emphasized that we ^re not trying* 'to 'evaluate any of 
your local reports or plans. Thus your 'actual' answers to the Guide- 
lines questions are* not our interest, ' although we hope they will be 
helpful .to you in either planning or reviewing an evaluation. We fc 
are simply looking for feedback from you concerning how useful w 
these Guidelines are in your own local, situation "and how we 
* might ma>ke them better. . ^ . v 

. ' ' - - " - SV " * / 

As you work through the Guidelines ijf terms of your local evaluation, \ . 

please feel* free to write any notes, ot ^comments 'directly on any ^ 

questions or sections that are unclear, ambiguous,' incomplete, etc. 

Such comments' will be of" great value. to us when we revise and finalize i 

the draft. After you have completed work on the Guidelines, please x_ 

complete the "Comments § Reactions" pageg at the end of this document. 

Then return your comments and the annotated Guidelines to 
» •» 

• Plsasfe rettirn them by 4 , * «■ ' ' 

Many thanks for your assistance with this project. * 
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j J.-ft.A. GUIDELINES' FQR REVIEWING EVALUATION REPORTS 



The purpose of these guidelines is to" assist individuals in fche analysis 
of reading program evaluations. It is important to realize th£t not all 
sections of this document will necessarily apply to a given- evaluation. 
Conversely, pertain 'evaluations might include content not addressed by these 
guidelines. It Is important, however, 'that the reading pnogram being 
evaluated is .described in terms of the program scope and sequence, and also 
in terms of its key elements. Inherent in this description would be' a 
sfatqja^t of the: , • x 

Vp^t?pose(s) of the evaluation r 

i K * * - 

- foqus of the evaluation 

local conditions or context In wl^ich, the evaluation occurred 

> ♦ * 

- procedures followed in conducting the evaluation 

7 ' * 

- sources^ information used in the evaluation 

%r 

" cpnclusions/recoimnendations based on the evaluation results 

* » 

* » 
It should be noted that responses of "No" to questions in the guidelines 

do not imply weaknesses in the evaluation being reviewed. Even in excellent 

and ^comprehensive evaluations,' many, questions might be answered "No", depending 

on the purpose and scope of the gi^an $v*tluation. y~ 



t Meaningful evaluation always results in the improvement of instruction 
fo,r students. It is essentially a plan of /for 'action that gives perspective 
to ttie curriculum., * 



International Reading Association.- Study of Raiding Program Evaluations Committee 

Michael Beck (Chair), Rebecca Barr, Roger Farr, Frank Ferris, Jerome Harste, 
Edward Smith, Susan S tuber — 



I. PURPOSE 

The report should provide the' reasons for the conduct of the evaluation'. Included shottd 
be a statement of why the evaluation was be 4 "" — j j-^j-j — ^. . , • 

ability and the purposes of- the study. 



-Is (Are) the reason(s') for conducting the evaluation 
explicitly stated? 

-Was the purpose recorded before the evaluation 
^ ■ took place?, x 

-Was the purpose communicated to various con- 
cerned groups prior to the "evaluation? 

-Does the 2^ort%ndi£ate who decided' the evaluation 
was advisable? . 

'•Is the degree to which various affected groups 
agreed with the purposes) of the evaluation dis- 
cussed? ' m * • 
i 

-'Are the probable outcomes of the evaluation suggested? 

)' -Is there an indication- of what actions might be 
taken based on^the various possible /outcomes? 

-E^pes^the report 'indicate that these possible 
actions ,are realistic and practical? 



Planning 


Report 


• Not 
'AppUc 


'YES 


NO' 


YES 


NO 








t 


* 










5 






9 








> 
















* 






























3 
1 



\ 



A 



Lb 



3. 



II. FOCUS 



The project, program, or the materials being evaluated should be described clearly and 

2? +™ 1S d «T5 ti0n Sh ° Uld include ««* el ^ s « the philo^phy ScluSni 

goals, objectives, and definition of reading; as well as spec" 4 " ---- "™< f MCAuaini 

students, personnel, instructional' methods , and materials. 

-If a "reading program" is being evaluated, does the 



tM ^^ vc , ^ u cvwu , utner material 

personnel, methods, etc.) of the program? 

-Is the "program" defined or explained? * ' 

-Is the primary focus of the evaluation clearly 
presented? 

-Does the evaluation state what aspects or p6r- 
titjas of the program were not" studied, and why? 

m -Are reasons for selecting these aspects and not 
others explicitly stated? 

-Are different 'evaluation procedures applied ,to 
various portions of the program? 

rDoes the report indicate the time span of the eval- 
uation in relation "to the time span of the program 
•described? 

• * 

* -Is a reason given £ot> conducting the evaluation 
, • during this particular time span? 

-Is the particular philosophy o£ reading instruction 
,embodie4 in the. program, stated* For exapple: Is 
reading seen as the recording process o£ turning 
graphic symbols into the sounds of 'our 4 language? 
Is reading seen ^s the acquisition of a set of 
identifiable but discrete skills where mastery of 
Itfwer-order sljills is seen as pretequisite^tb mastery 
of higher-order skills? Is rfeeding seen as a whole- 
language process in which meaning is central and ♦ 
particular skills a?e never isolated but become 
important- only as they relate to this focus on 
meaning? , % 



. Planning 


Report 


Not 


YES 


Na 


YES 


NO * 


\pplic 
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j 11 « * i « 

A» ruNDING . .» 


Planning 


Report 


« no u 


-Ha3 the school -district examined, the funding pattern 
cff its reading program to determine the sources of its 
funds? (Funding. sources cfcujd include local dollars, 
f xompensatoiy^educatroh fiinds, grant money, alloca- 
tions from state and ffederal' agencies', etc.]f 


YES 


NO 


YES 


NO 


Appli^ 












* . j the district* determined* where and how those 
fimds are spent? (Basal series, support mate- 
rials, salaries' of reading teachers, staff 
development, activities, 1'ibrarjr books, building 
consultants , *etg . ) 




t 




i 

V 




^ » 
B. SITES : * 










• 


-Are the sitc^s (buildings or^ classVoom^in which the 
^ evaluation took place described (in terms t>f number, 
representativeness, reasons for selection, etc*)? 












.-Is the "program" essentially* identical* across 
sites? 

* 










) 


> 

-If not, is^the evaluation separated by site? 










j 


-Are the space/sizs and physical arrangements of 
0 the sites and their potential effect on the pro- 
* / gram considered in the evaluation? 






1 







C. STUDENTS 



-Are relevant characteristics of the students _ 
volyed (e.g,., sex, age, entry* capabilities, language 
diversity, past learning v rate) described? 

-Are these characteristics analyzed in the 
^valuation? ( 

-Is the stability of the group over the fSeriod 
of the evaluation described? 

-If more than one site is evaluated, are students ' 
across 'the different sites "eou^'arable" in -terms 
of relevant variables? 

-Have "fhe students previously been .involved with 
• this program? 1 , / 

# '* ^ & 

-Was feedback provi s de4 to students concerning the* 
evaluation? * * \ 

-Are feedback procedures\ described? 











* \ 
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Planning '* 


Report . 


Not 




c ' 

-IS the student-to-teacher or student-staff ratio 
included? ' 


YES 


NO . 


.YES 




Appli^ 




— ■ \ 


s ■ ■ 









D. STAFF 



-Ate the general characteristics (number, role, age, 
education, special training, experience, etc.) of tl 
staff presented? 

-Are any of these factors^ analyzed as part of 
the evaluation? 

-Is the f, managerial ff or organizational scheme from' 
teacher to supervisor fresented? i 

t -Was this sqhemepart of the evaluation? 

-Is the staff r6ie in selecting .and implementing the 



s schem 
r6ie i: 

reading program described' 



-Are staff development activities based on relevant 
research and the identified needs of various levels 
of personnel described? 

• * ■ • 

-Are criteria for determining staff development 
activities for the district stated? 

. ' \ 

-Is the role of the staff in the selection of 
staff development activities assessed? 

-In* the evaluation of staff development 
activities? ^ 

-Is staff * self-assessment part of the overall evalua- 
tion? , 

\ 

-Was feedback provided to staff concerning the evalua- 
tion? ■ <. 

-Are feedback pr^aedures described? 

INSTRUCTIONAL METHODS # — , 

, « 

-Is there a description of all instructional methods 
being evaluated? 

x ' ~Is^a rationale" given for the~use of .those 
particular instructional methods? 

-Is an attempt made- to match diagnosed student 
needs to appropriate instructional methods?- 
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-Are procedures* for selecting methods to 
correspond to students needs, assessed? 
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-Is .the degree of correspondence between the instruc- 
tional .methods used and the district's philosophy of 
learning tp" rea£ described? 

Were a variety of instructional methods' used 
with students? . ■ 



MATERIALS 

-Are the materials and their important characteristics ' 
te.g., length, difficulty/ format, * previous use) 
described? _ 

-Are the procedures and personnel involved in develop- 
ing or selecting the materials indicated? 

-Were the materials developed for use in programs 
similar to the one being evaluated? 

-If not, is a rationale provided for their use? 

-Were the materials used 'essentially in their entirety? 

Jj^ *Were changes or adaptations made in the materials 
for program use? . ^ 

* -Is' there a description given of* how the portions' 
to be used were selected? 

-Are previous evaluations of these materails in * 
" similar settings available? 

-If sjp, are they summarised or referred to? 

-Is the relationship between instructional materials 
and strategics "used and the district's reading pro- 
gram goals described? 

-Has the appropriateness of the materials to the 
goals of the program been jdetermined? 

-Are the procedures used in the making of 
this determination di$cus£ed or referenced? , 

-Are procedures described for selecting materials to 
correspond to student needs*? 
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III. LOCAL CONDITIONS 



^qeptxonal conditions occuring during the evaluation period should be carefully described 
so the findings of the evaluation can be interpreted in light of them. Such internal 
oX^^ h ena M ie th6 ^ reader t0 the'findings Sight be S in 

of^iSStSi? Pr ° V " liMtati0ns t0 the gen^ ralizability of the results 



-Are any exceptional .local conditions described that 
might have affected materially the evaluation? 

-Did the context of the evaluation iWide a true 
picture of the -reading ^program? ' f 

-If not, was this accounted for in describing 
the limitations or interpretations of the study? 

-Were staff members and students aware of the evalua- 
tion while it was being conducted?' 

-If so, is th^rer reason to believe such aware- 
ness may have Significantly altered, the evalua- 
tion? 

-Were the significant internal "political" factors 
impacting the program at the time the evaluation was 
conducted? 

-If so, are the results of ttfe evaluation dis- 
cussed in terns- of the potential impact of these 
factors? 

-Were there any special personnel factors 'that 
JRightr have affected the evaluation -- e%g., a 
teacher strike, staff firings, student problems, 
new key administrators? 

t < 

-If so, is the potential impact of these factors 

discussed? * 

** * 

-Were there, jny^pscial physical factors that might 
have .affected the evaluation ~ e.g., facility 
problems, late delivery of materials, routine 
start-up problems? 

-If so, is the potential impact of these factors 
discussed? 

• Were thera any important social or community factors 
that might have affected the evaluation — e.g., 
school bond issues, exceptional local economic con- 
ditions, sch6ol legal issues? 

- -If sq* is the potential impact of those factors 
■ discussed? 



Planning 



lis 



NO 



YES 



Report 



NO 



Not 
"Applic 
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IV. PROCEDURES 



C 



r Is the design of the evaluation described in 
sufficient detail- such that v the study could • 
essentially be replicated, given only the informa- 
tion pu^ented? 

, -Are limitations of*the procedures discussed? 

-Did the evaluation inclyde an analysis of special 
♦subgroups- of \ students ^.'g.> control and experiental 
groups, Title I arid non-Title ^students)? 



-If so, is the evaluation designed to determine 
whether any results are a direct reflection of 
the reading program? 

-fyould Similar changes have resulted with nq 

special program or from the instruction normal 1) 

<. provided? 
* 

-Is th'iTs possibility controlled for or dis- 
cussed in the report? ^ 

-Is the evaluation designed to isolate, key components 
of the reading program for .further analysis? 

Jfas a timeline for the evaluation established during 
' the planning stages? 

-If so, is evidence presented thajt the timeline • 
was followed, or are significant^ departures 
noted? 

-Is (Are) the person(s) who conducted the ^valuation 
identified? 1 # 

-Is a clear description of the role and duties of the 
evaluator(s) given? » 

-Are reasons presented -for the selection of the 
evaluator(s) presented? / 

-Did staff have a role in selecting the evaluators? 



-Are th£re other dati indicating support for the 
•evaluation? 



-Are the evaluators* bi&ses (methodological or 
\ instructional) noted? , 

\ 

\ -Is there evidence that biases are controlled or 
\ ■ 'minimized? 

\ 
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Report 


Not 


YES 


NO 


YES 


NO 
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IV. PROCEDURES (Continued) 



4 



Classify the evaluator(s) as: Internal (permanent district' employees) 

External (outside consultants) - 
. "Mi'xed" ' (Internal and External) 



Now complete the appropriate sections below; Part A for Internal evaluators 

Part B for External evaluators 
Part A $ B^for "Mixfed" evaluators 



A. INTERNAL. EVALUATORS 



-Are the* evaluators ?taf f members of the program 
being studied? 

-Are their previous experiences with similar 
types of evaluations described?. 

-Is evidence presented concerning their 
technicaf expertise (in evaluation and 
in reading)? 



-Is evidence presented concerning the 
ability of the person (s) to evaluate 
objectively the reading program? * 



B. EXTERNAL EVALUATORS 



-Has the person had previous experience evaluating 
programs of a similar nature? / 
& » 
-Is there evidence of qualification in terms 
of knowledge of reading, instruction, and 
evaluation? 

» -Is the evaRiator accessible for follow-up 
consultation? 

-Is an explanation given of why this person 
was selected for the project? 



-Does the report indicate who within 
the district had a role dn choosing 
this evaluator? 

-Was the amount of time the person spent in the 
district studying thtf program specified? 



Planning 



YES 



NO 



Report 



*YES 



NO 
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V.'. INFORMATION SOURCES 



.The evaluation should describe as completely as practicable .sources of the information 
used in the report. Included should be a description .of the various, sources of data' 
used m the evaluation, the characteristics of these instruments or procedures, and why 
they were included. • •• • v * 7 



A. DATA COLLECTED 



•Are interpretations in the evaluation b^sed on 
multiple sources of, information — both quan- 
titative and qualitative? t^-gv , attit.utfe, 
achievement, ability test results; interviews; 
questionnaires, rating forms) 

-I? quantitative evidence summarized Clearly? * 

^Is the meaning of any statistical tfests 
discussed? 

-Is educational significance of the result^ 
discussed? 



/ 



-Are qualitative data included? 
-Are they summarized clearly? 



-Are conclusions based solely or -primarily on 
single sets of data? 

-Do other data support the conclusions? 

-Are any longitudinal data provided to < 
support the conclusions? . 

'-Ate suggestions made for follow-up data 

J collection collaborate the results of to \ 

\ analyze long-terra trends? ) 

-Were ^iata, collected from all involved groups 
(students, teachers, parents, various admin- 
istrators, board of education)? ^ 

-Vfere data gathered as near the beginning and 
end' of the period to be, evaluated as possible? 

-Would additional data concerning leaning or 
attitudes — of students or professionai^-staff 
have been impo^t^nt to the evaluation? 

-If achievement tesV data ate included/ is a - 
rationale given for the specific «test that was 
used? ^ 

2 b 



Planning H 


Report 


N<?t ^ 


YES 


NO 


YES 


- NO 


Applic 
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d3tA COLLECTION (Continu 1 

- -Is evidence provided that the test content 
, matches that o£ the instructional program? 

* 

-Is this test series typically used -in the 
system? « s t 

/• 

-If not, Is, the reason for its use in the 
evaluation presented? 

-tfere^data analyzed and/ or reported in -terms of 
z the! important purposes of the evaluation? 
f 

-Was the "level at which results were 
summarized consistent both with the 
purpose} of the evaluation and the 
4 instruments included? (e.g., a • 'reading 

comprehension" test was used, but results 
were applied to a phonics, instructional 
prograto.) » 



;Are summaries provided of the results of each 
set of information collected? * 

♦ -Is some evidence provided of "quality control" 

in the collection of the data? 

-In the analysis or processing <f the data? 

e 

TECHNICAL CHARACTERISTICS ^F THE DATA 

* Validity - 

-is^a rationale provided for the selection and 
'use of the instruments included iri tK§ evaluation 

-Is some ddcumerf|&on giveif ^for the validity of 
<he instruments usm for the? purposes they 
serve in the evaluation? 

. -For achievement measures, is 4 evidence of 
content validity Xappropriate match of pro- 
gram and test rontons) ^trovided? 

•Portability or aptitude measures , is there 
• evidence or criterion or otherv appropriate 
validity? . i . 




•For other measures (e.g. f% qu^tio^ires, 
surveys, attitude instruments), & some 
evidence of 'validity or sound instrument 
construction procedures presented? 



TECHNICAL CHARACTERISTICSfOF THE DATA- 



Validity (Contipued) 



-For instruments constructed specifically for 
n this evaluation, is there ^/idence of any 

validation or tryout procedures of the measure 

during its construction? 



-Was there a pilot testing or tiyout of 
such instruments, before their use in the 
program? 



Reliability 



-If any instruments used were constructed 
^specifically or modified significantly for this 
evaluation, is reliability evidence presented? 

-If not, -does the construction (question 
clarity, length, required tasks) suggest 
thfet the information obtained is reliable? 

--tferd reliability data collected as part ,of the 
evaluation or in a separate study? 

• * 

-If in a separate study, is there evidence 
that the setting and groups used for cal- 
culating reliability were similar to 
those of the evaluation? 

V 

-If published instruments were used, are reli- 
ability data available? * 

-Were the data gathered on grqups comparable 
to those being evaluated? 

6 <> >- 

^If not, were reliabilities provided 
for the evaluation sample? 



J 
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Not 
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YES 


NO 
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NO 
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13. 



J 

e in the ^valua- 



VI. CONCLUSIONS " 

^nn S T! a ^ 6f th f *f aluation "P° rt should present the assumptions mai _ _ ™ 
sets J? dSj l°n ?rj ea ^ d ; SuCh conclusions need not all be supported by alj 
!Se!« S«SJi S Nevertheless, conclusions should be referenced to the results 
P° ssi ble. Conclusions not supported by the data should be clearly identified 

W«Stfnf 2 rePOrt Sh f " C ° nClude a pr °P9 s *d P ian of actio * for eiSer 
implementing the recommendations or initiating appropriate follow-up action. 



-Does the report contain a complete list of ? 
assumptions made in designing and conducting * 
the evaluation? (e.g., it-is assumed that the • 
XYZ Reading Test is an adequate and complete 
measure of comprehension.) v 

-Does the report clearly' state the conclusions 
reached? 

r Are the conclusions reached directly tied 
to the, results obtained from the evalua- 
tion? * 

-Are the evaluation results that yielded 
inconclusive findings clearly indicated? 

-If generalizations or interpretations are 
made that are not based on soMd data, *re 
they clearly laftled as such?" ' * ** 

-For each concfusion drawn, are the supportive 
evidence — qualititive and/or quantitative — 
for this conclusion referred to? 

-Dd the conclusions follow naturally from 
what was actually measured? 

-do the conclusions include tHe reasoning 
that led to them? 

-Are inconclusive areas indicated and 
reasonable interpretations provided for 
these? , * 

-Ails other reasonable interpretations 
provided? 

-'Are limitations of the evaluation presented 
frankly? - 

-Are these conclusions stated in a manner that - 
leads ta a plan of action? 

* , 
-Is evidence available to shovf that a relevant 
action play has been established and put .into 
practice? ' * 



Planning 



YES 



9 



NO 



Report 



YES 



NO 



Not } 
\pplid 
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-VL . CONCLUSIONS (Continued) 



-Is program sta£f aware of the intended 
implementation of any findings including 
new goals, programs, ^procedures, materials, 
appropriate timelines > staff development 
activities,, etc. 

-Are conflicting poiots of view concerning the 
program and the evaluation results presented 
fairly? 

-Are "mijic>ritx"* vTews represented? 

-Does the report explain why these views 
are not consideVe4 "correct," without 
attacking thes^ views in a non-objective 
manner? * . £ 

-Is some assurance given that the report was not 
modified against the best judgment- of the m 
evaluator to reflect the views of school staff ^ 
with vested interest concerning the progTam? 

-Have the evaluators analyzed and interpreted* 
specific results in light of the broader pert- 
aspect ive' of reading^ and^ 'th$ teaching of reading? 

•*Is there a plan" for the dissemination of informa- 
tion derived from the evaluation? 



Planning 


Report 


Not . 


-YES 


NO 


YES 


NO 


\pplic 






✓ 
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.APPENDIX^. Summary of Comments on Field Test Guidelines by Local 
t. District Reviewers (N = 19.) „ * v. 

^ * 'Name: j 

( (OPTIONAL) * ~ 

c Title: 



'COMMENTS a REACTIONS 



I. In order to be useful for different types of evaluations, these Guidelines 
- must be clear, concise,, and comprehensive. Please give your opinion on the 
following points. Feel fr^e to suggest changes or additions or deletions. 

Very Somewhat 



No 



A) 


Are the Guideline questions* readable? 


. 5 


13 


B) 


Is the intent of the questions clear? 


6- 




O 


Is the organization of the topics clear? 




2 


D) 

E)' 


Ajre the individual Guidelines; sufficiently non-technical? 
Are the criteria r«*alistic^io*r a typical school evaluation? 


* 


?. i 


F) 


Are the Guidelines comprehensive? 


:8 


* « ■* 


F) Is the coverage too broad for a tvpical school PvalnaiHon? 
Please comment on any of the ahove: ' 




/ — ' 










: 1 ^ . j 


— T i 




f w . 







II. Indicate below your judgment of the usefulness of the' Guidelines in your 
Own situation. ^ 



A) .To what extent would you find- the Guidelines 
useful for: . t * 

1. Planning an evaluation? 

2. Conducting an evaluation? 

3. - 'Interpreting an Evaluation? * 



Very Adequate Adequate Inadequate 



Comments : 



r 



(■ ■ ■ • 

B) Rate each of , the major sections of the Guidelines £n 
tenns of^its usefulness in plaxining or interpreting 
an 'evaluation report: < t < 



Very Adequate' Adequate Inadequate 

' " 15 2 ' " 



I) Purpose - \ 

II) Focus , % ■ 12 c ' 

III) 4 Local Conditions 



IV) Procedures % , 

V) Information Sources' 
Vp* Conclusions 



6 
11 ' 



* ' 12 . . 5 



* 4 • » 

Strengths provides a good base; thorough; Sections I an d II are excellent; 
- veil -organized;.^ relevant-; very systematic' 



'• : — y - 

Weaknesses: language and wording; techn ical; time-ccnsuming; criteria toe broad; 
too cpmplex for typical school; neglect of "teacher/process variables 



IV. Can you suggest any additional questions or iaroortant areas that should be * 



included in the Guidelines? 

Needs section on use of accented nractice^' 



Needs more examples . . 



r ollo^-Up should be included * n 



How are student needs assessed 



V. As # a result of using these Guidelines, specific changes, if any, would 

you make in any future evaluation acti^Lti^s in which you might participate? 

Look at more aspects of the reading "urogram" 
: : " S 



Better dissemination procedures 



Demand a more thorough and systematic .job by the outside group — now I >nov what to 

— ■ . — 1 r.r.'rr f^r in a good Job 

Set more input from all involved groups * * 
' 1 : — ■■ u-l 



THANK tOU VERY MUCH *0R YOUR COMMENTS , 



* f 



.* • • * . 7/81 Draf 

APPENDIX C. Draft of Final IRA Guidelines 



I.R.A. GUIDELINES FOR REVIEWING EVALUATION REPORTS . 
> ' ■ ■ 



The purpose of these guidelines. i& fi# assist individuals in the analysis 
of reading program evaluationsT^rT is important to.realize that not all 
sections of thife document 'will necessarily apply to a given evaluation. * 

_ * , > 

Cqnversely, certain evaluations' might include content not addressed by 
these guidelines. It is important, however,, that the reading program 
being evaluated is described in tenns^of the pyogram scope and sequence, 
and also in terms of its key elements. Inherit in* this description . 
would be a statement V of the: > j 
r purposes), of the evaluation* 

- focus of the evaluation . s 

- procedures followed in conducting the evaluation 

- s^urfces of information used in the evaluation 

- conclusions/recommendations based on the\ evaluation results 
w* / T 

Responses of "No 11 to questions in the guidelines do not imply weaknesses 

in the evaluation being reviewed/ Even in excellent and comprehensive 

evaluations, many questions might be answered ,f Nq n , depending on the - 

purpose and scope of the given evaluation. 

% » 

r 

Meaningful evaluation always results in the improvement of instruction 
for students. A useful evaluation is essentially a plan for actiofi that 
gives perspective to the curriculum. * 



Int ernat Zonal Reading Association — Study o f Reading Program Evaluations 
Committee 

Michael Beck (Chair), Alden, Moe (Board Liaison), Rebecpa Barr, Roger Farr, 
Frank Ferris, Gerald Maring, Edward Smith, Susan Stuber 
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I. PURPOSE 

The report should provide the 'reasons for the conduct of the evaluation. Included 
should be' a statement of why the evaluation was conduc 
advisability and the purposes of the study. 



A. The reason (s) for conducting the evaluation is (are) 
explicitly stated. 

* ■ -The purpose was recorded* before the evaluation took 
place. 

-The purppse was communicated to various concerned 
groups prior to the evaluation. • 

B. The report indicates who decided to conduct the 
evaluation. 

C. Probable outcomes of the evaluation are suggested. 

. -Action' that might be taken, based on the various! 
possible outcomes are indicated. 

t 

-The report' indicates that these possible a'ctions 
are realistic and practical* 

( 



NOT \ 
• YES . 'NO IAPPLIC. 








i 


% 






















* 
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T 



3. 



II. FOCUS 



The project, program, or the materials evaluated should be described clearly and 
completely. Hie description should include* such elements as the philosophy", . ' 
including goals, objectives and definition of reading; as well as specifics such 



The primary focus of the ^valuation is clearly presented, 

-If a reading program is being evaluated, the evaluation 
focuses on all important aspects (text , personnel, 
methods, etc.) of the program. 



, — ~~ 1 7 — — 

all components. 

-The evaluation states what aspects or portions of the 
program were not studied, and why. 

-Different evaluation procedures are used with various 
portions of the pijogram. , 4 

B. , A- reason is given for conducting the evaluation during 
this particular time span* 

C. The particular philosophy of reading instruction 
embodied in the program is stated. 

D. FUNDING ~* 

The funding *pat tern of the reading program as been examined 
4 to determine the sources of its funds. (Funding sources- 
could include local dollars, state/federal allocations , etc. ) 

-The evaluation includes an analysis of where and how 
funds are spent (e.£., bas^l series, support mate- 
/ 'rials, salaries, staffs development activities, library 
books, consultants). 

E. SITES * 

* 

Building or classroom sites in which the evaluation 
took place ^:e described in terms of number, representa- 
tiveness, rAsons for selection, etc. 

-The "program* 1 is essentially identical acVoss .sites. 

-If not, the evaluation is separated by site. 



materia 

YES 


Is. 

NO 


NOT 
APPLIC. 






















• 


- 






> ■ 










• 










• • 










v. 

i % 
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f. L£CAL CONDITIONS 

Exceptional conditions occurring during the evaluation 
/period are described §o the findi ngs of the evaluation ' 
can be interpreted, in light^of them. .Such descriptions 
should enable- the reader to judge whether the findings < 
might be similar in-*>ther contexts and' provide clear 
limitations to the generalizability of the results to 
other" situations. 

The setting in which the evaluation took place provided ' 
a true picture of the reading program. 

-If not, this was accounted for in describing the 
limitations or interpretations of the study. 

Were staff members and students aware of the evaluation 
while it was being condtfctedZ 

-If so, might this awareness have significantly 
altered the evaluation? 

Any exceptional local conditions that might have affectfed 
the evaluation are descried. - (E^g. personnel factors 
sxxch as strikes or firings; physical factors such as facility 
or start-up problems; or social/community factors such as 
bond issues or exceptional economic conditions^) 

i 

-The potential impact of any of these personnel, physical 
or social factors is discussed in the evaluation. 



G. STUDENTS 

> 

Relevant characteristics of the students involved as sex, 
age, entry capabilities, language -diversity, e±c. are 
described ♦ 

-These characteristics are analyzed in the evaluation. 

The stability of ''the group dver the period of the evaluation 
is described. * 

Students across tjie differentia it es are "comparable" in terms 
of relevant variables. 

-The method far determining comparability is described. 

V t 1 
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H. STAFF '£ ' 

The number, role, education, experience, other relevant 
characteristics of the staff a;e presented. 

The "managerial 11 or .organizational scheme from teacher 'to 
supervisor is presented. 

The staff role' in selecting and implementing the reading 
program is^described. 

Staff development activities were provided for personnel 
^involved .with the program, " 

-The role of the staff in the selection of staff 
development activities is discussed.. 

Feedback was provided^© staff concerning; the evaluation ; 

-Feedback procedures are described, j 

I. INSTRUCTIONAL METHODS 

Are instructional methods bfeing evaluated? 

■ * 
-If *so, these methods are described. 

•r* 

tA. rationale is given for the use of those particular 
instructional methods. 

-An attempt is made to diagnose student needs and 
match students to appropriate instructional methods. 

A variety of instructional methods were used with students. 

-The correspondence between methods used and the 
district' 8 philosophy of reading is described. 
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J. MATERIALS 

* i 


j 

YES 


NO' 


NOT ' 
APPIsiC. 


Materials used in the program and their important character- 
istics (e.g., length, difficulty, format, previous use, 
commercial/unpublished) arp described. 








The procedures and personnel involved in developing or 
selecting the materials are indicated. 








i • * 

The materials used were developed for use in comparable^ 
situations and with comparable student groups. 




x 




-Any changes, adaptations, or use of selected portions * 
of the materials are described. , 








^Previous evaluations of these materials, if any, in similar [ 
fc settings are summarized or referred to. 








The relationship among instructional materials, strategies 
used, and the district's reading program goals is described. 




0 




-The appropriateness of the materials to. the goals* 
tff the program has been determined. 


* 






-Procedures used in making this^determination are 
discussed or referenced. 


— ' 








* 




i 

i 


• 

» 


t 






4 

• 


• 


- 


« 






) 










- 


J a 




■ 




* * 
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III.. PROCEDURES 



The procedures and/ Qr experimental design used in the evaluation should be discussed 
in enough detail so readers can judge the appropriateness of the conclusions reached, 
^valuators should be named, along with a description of how they became associa&d 
with the evaluation and their relevant background and experiences. 



-4? 



A. The design of the evaluation is described in sufficient 
detail to enable its replication. 

-Limitation's of the procedures are discussed. 

B. .The evaluation sine ludes an analysis of different groups 
of students (e.g., cbQtrol and experimental groups, Title, I 
and non-Title I students 

-Would similar changes have resulted in the absences of 
^ the special program or as a result of the instruct- 
ion normally provided? 

-Is this possibility controlled for or discussed in 
the report? 

C A timeline for the ev^lufc.tion was established during the 
planning stages. 

' -The timeline was followed or significant departures^ 
are noted. „ 

D. The evaluator(sr) is (are) identified. 

■ -A clear description of the role and duties of the "V. 
evaluator(s) is given.' 

* . 
—Reasons for the selection of -the evaluator(s) are 
given. ' * ' 

,-The evaluators 1 biases (methodological vor ' 
instructional) are noted. 



• not; 

YES 1 NO 1 APPLlC . 


\ 




















" 


- 
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8. 



III. PROCEDURES (continued) 



Classify thfe evaluator(s) as: Internal^ (permanent 'district employees) 

* External (outside consultants) 

• "Mixed" * (internal and external) 



* Complete the appropriate sections below: Part E f or Internal evaluators 

9 Part F for External evaluators 
Part E & F for "Mixed" evaluators 



E. INTERNAL EVALUATORS 



J 



Are the evaluators also staff members of the. program 
being studied? 

* 

Their previous experiences with similar types of 
evaluations are described • 

-Evidence is presented concerning their 
technical expertise (in evaluation and in 
reaaiag) . 

- -Evidence is presented concerning their abilijty 
to be objective in tfreir e,mLuation of the 
program* * 

F. EXTERNAL EVALUATORS 

The evaluators have had previous* experience studying 
programs of a similar nature. 



-Evidence of expertise in reading, instruction, and 
evaluation is presented, 

-The evaluators are accessible for follow-up • * 
consultation. 

t 

' -*fin explanation is given of why the evaluators were 
^selected for the project. 0 . 

-The report indicates which district personnel 
had a role in choosing t;he evaluators. * 

-Data indicating staff and administrator support 
for the*evaluator are presented. - * . 

The amount of time the evaluators spent both in the .district, 
and in total studying the progranr is specified. ^ x 

\ . . 



YES 


*• NO 


NOT / 
APPLIC.*" 












1 * 












J 




' / 






l 




• 


1 




V 
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9. 



tV. " INFORMATION SOURCES, ; ^ \>#- • * 

The report should describe as conq>le't:*ely.a£H'tacticai the' sources of the information 
used in the eyaluat-ion. Included srhou/Ld be a< description of the data used, the 




y A. DATA COLLECTED ^ ,' -* • 

Interpretations in the evaluation are based on ■ 
multiple sources of information — both n qu£n- 
.titative and qualitative (e.g., at^fcitude, 
achievement*, ability test results; interviews; ' 
questionnaires; rating forms). 

% , * '-Quantitative evidence is summarized clearly. 

* -The meaning of any statistical 'tests is 
, ' * . discussed. 

-Educational implications' of the results , 
A . 'aradiscussed.* • 4 



-Qualitative/data are included, 
• • -They are ,sutmaat£ized clearly. 



-Conclusions are based solely or ^primarily on 
-single setg x^f data. 

Suggestions are made foy follow-up, d^ta collection 
to collaborate the Results or to/analyze longrteral 
, trendy. 

<Data were col^Wct^d fromlall" involved groups (e.g?, 
students, teachers V p^rentsT^arious administrators, 
^boara of education), 



y 



The time at 



were collecte^Ls spdcifi#d.. 



.Thte ma&er in which results are/sqjmnaitz^d is * 
; consistent both with the purposes of the evaluation', 
and the instruments included (e.g., a "reading* * 
comprehension" ,test was used*, but results were 
applied to a phonics instructional program) . ' 

« •» m ' * 7 • * • 

Summaries are provided of th©Lre/ults*of ea'ch set 

"of information collected. * * A 

AY* Evidence is 'pratfided of "quality contfrol" in the 
W collection of the data* J* 

-<tn Jthe an^l^ais or processing of the data. 

ERIC . • * ' * " . . / 41 



S 

7 YES 


NO 


i NOT. . 
5CPELIC* J 


* 


m 


* 














« 












; m 












1 

V 








M ■ 


i. 


— a 




*-< 

■/ 












* 




T 









B. TECHNICAL CHARACTERISTICS OF THE DATA 

t 

Validity ■ . 

A rationale is provided for t^J^section .and .use df . 
'the instruments incpLuded in the evaluation, 

P\\ ; £, 

Documentation is given of the validity of the * * 
instruments used for, the purposes they serve in % „ 

the evaluation. * < * 

» * * 
-If an achievement test was used, evidence is 
provided, that the test content closely matches 
that of the-instructional program. 

The test .series usdd in the evaluation is typically 
used in the system. * # 

-If not, reascms for its use in the evaluation 
are specified. 

* . 

For locally developed 'instruments, evidence is provided 
of validation procedures of the measure during its 
* construction. > * «- • 4 * * -B 

-Sufh instruments weris* pilot tested before their 
upe in th^ program. \ 

: . Reliability . 

For instruments ttsed that were constructed specifically 6 
'modified significantly for this evaluation, reliability 
evidence* is* presented. ' * ' 

■ * ■ 

-If not, does the construction (question clarity, 
length, required tasks) suggest that the informal 
tioji detained *is reliable? 

^ If published 'instruments were used, reliability 
^ data are reported. ^ 

-D^ta were gathered on. groups comparable *tQ 
those b§ing evaluated, -w 

\ -If $ot, reliability data are provided for 

y J thp sample of students used in the evaluation 

'•-§;• •' • • 4? • ' 

. - ^ ■) 
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V. CONCLUSIONS 



The summary should present the assumptions made in the evaluation and £he 
conclusions reachedV^Conclusions need riot all be. supported by all sets of 
data collected; nevertheless,, conclusions , should 'be referenced to the results 
wherever possible. /Conclusions not supported by the data should be clearly 
identified/" Finally, a good report concludes with a proposed plan of action 
*for either implementing the recommendations or initiating appropriate follow- 
up action. « v 
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The report contains & complete list of 'assumptions 
toadein designing, and conducting the evaluation 
(e.g., it is assumed* that the 'XYZ Reading Test is 
an^iequate and complete measure of comprehension). 

r 

The report clearly states the conclusions reached. 

-Conclusions reached ^are limited to the 
results obtained fr^m the evaluation. 

* * - 

-Results that yielded inconclusive findings 
are clearly indicated. 

f -Generalizations or interpretations, made that 
h are not" based on solid data, are dlearly 
labeled as such. * f 1 

For each conclusion drawn, the supportive evidence — 
qualitijj^^j^gQgr quantitative — is referenced. 

* -The .conclusdta*. follow from whst was 
actually mea^i^ed . 

-The conclusions include the reasoning *thatf led 
to them. 



-Inconclusive ajeas a*e indicated and" reasonable 
interpretations provided for these. 

-Limitations of the evaluation^ are presented, 

-Conclusions are state^ in a mariner. tfyat leads 
to a plan of action. 



-Evidence is provided to show that a relevant plan 
^of actitin h&s been ^established and implemented.- 

\ -Program sGaff is aware of the intended implementation 
, of any findings including new goals, procedpres, 
• * materials, timelines^ activities, etcy 

# Goriflicting' points of* view concerning the program and 'the 
M <»jaluation results are presented fairly. " " 

er|c ' 
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V. CONCLUSIONS (Continued) 

-"Minority 11 views are represented % 

'-The report presents evidence as. to why 
these views are not considered "correct, 11 



without attacking these views in # a 
objective manner. 



Some assurance is given that the report was not 
iaodified to reflect the viey^, of ■ school staff 
having vested 'interest concerning the program. . 



The is a concrete plan for tie dissemination 'of « 
results of the evaluation tcPall concerned, groups 
(e.g. , students, staff, ^tfre public). 

r 



ups 



YES 


NO 


NOT 
APPLIC. 






9 






W 








A 




♦ 0 



y 



■4 



ERJC 




APPENDIX D. Memo to IRA Committee Reviewers of Draft Final Guidelines 

INTERNATIONAL READING ASSCCIATiON t *300 3arKsciie Roas* 3g/=/^? \ewar* ^e'a*^ ■ rf SA 
ASSOCiATIQN JNTSHNATIONALE POUR LA L£CTURE * ~eiechO"e :02 ~C *6CC 

ASOCIACION INTERNATIONAL OE LECTURA t : ac . e ^eadmg Newar* Delaware 



September -2, 1981 



MEMO TO: IRA Committee Members 



* FROM: Michael D.^Beck, Chair^P" 

Committee for the' Study of' Reading Program Evaluations- 



RE: 



Guidelines for Reviewing'Evaluation Reports ^ 



r 

Over the past setfer^l months,* the Committee far the <Study of Reading 
Program Evaluations has been working on a set of "Guidelines" for 
reviewing evaluation reports. This frork, which included a field 
tes^/in^f pur. school systems, was supported by a grant from the Na-'- 
^£i<mal Institute of Education. The feedback we received from' the 
fiel#*test participants. was" used to revise an earlier draft of the 
enclosed Guidelines* 



We would greatly appreciate your assistance at this stage of our 
work. Will you please critically review the enclosed draft, making 
any comments or suggestion^ directly on the, document? Your sugges- 
tions might be of an editorial or wording nature, or more substantive 
— either are welcome* After you have completed the review, please 
fill out the short "Comments & Reactions? at the end of the document* 
The total amount of time this review will take from your busy sched- 
ule should not exceed 60 minutes* Again, fe6l free to respond both* 1 
on the pag^s of the Guidelines and on the "Comments & Reactions" 
section at the end. 

7. 

It is^ important thai we receive your reactions no later than Septem- 
ber 24, -so they can .be* taken into accounfe in our final draft. Please 
use the enclosed envelope to return the S&cument before that>date. 
Thank you for your valued assistance. 



MDB/ss 



j Enclok^res 



9 

:RIC 
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APPENDIX £. Summary of Comments on Draft of Final Guidelines by IRA 
Committee Members (N = 41 ) 

< Name :£ (Optional) 

• fcommfttee : 

r 



COMMENTS § REACTIONS 




* ■ 

In order to be useful for different types of evaluations, these Guidelines 
must be clear, concise, and comprehensive. - Please give your opinion on the 
following points. Feel free to suggest changes or additions or deletions, 

z • Very Somewhat No 



A) 


Axe the Guideline questions readable? * 


zi 


-•9 




B) 


Is the intent of the questions clear? 


21 


10 




C) 


Is the organization of the topics clear? 


32 


5 


1 


D) 


Are the individual Guidelines sufficiently non -technical? 


28 


9 


1 


E) 


Are the criteria realistic for a typical. school evaluation? 


13 


15 • 


3 


P) 


Are the Guidelines comprehensive? 


32 


6 


G) 


the coverage too broad for a typical school evaluation-? 


7 


18 


11 



Please comment on any 



Stylistic/ editorial /fomat suggestions ( 3 ) 



I 

Indicate below your judgment of the Guidelines' usefulness in a typical school 
situation* s 

Very Adequate Adequate Inadequate 
A) To what extent would* you find the- Guide- * j 

lines useful 'for: f "* 



, 1. Planning an evaluatipn? 0 27 ■ 11 Q 

2. Conducting an evaluation? _ 20 15 ~> 



3. Interpreting an evaluation? " Oo Hn • * ' 

Comments: is a S° od "Modfel" >6x an example of the "ideal" 

■ 1 , ; — ; ■ ■ 9 * 1 

that should be kept in mind while planning/executing an evaluatic 
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B) How might the practicality of the Guidelines for a typical school be improved? 

Reduce 1 the length (2) » ' 



Set /indicate priorities on the — list^e.g, "essential" , "suggested")- 

. . # * « 

C) Rate each of the major. sections of the Guidelines in terjBS of its usefulness 
- : ii^ planning or interpreting .an evaluation report: 

Very Adequate Adequate Inadequate 
• I) Purpose 18- 15 \ 

II) Fdcus' ^ i 8 15 

Itl) Procedures * * - 20 T3 1 ^ 

IV] Information Sources 00 in 

V> Conclusions • 23 ' 9 ' 



III. What would you consider the major strengths and weaknesses of the Guidelines 
in the|y current forptf 

Strengths: Comprehensiveness/spope (16) ' * . 

Clarity (U) " 



Weaknesses: Length (k) 



Format,* wording (h) 



IV, Can you suggest any additional topics or important ^reas that should be -included 
in thfe Guidelines? 

None (5) * ^ 

Individual comments and suggestions (5) 

7 : 1 : — 



V. Can you suggest any particular section of the Guidelines that should be 
eliminated, rewritten, or modified substantially? 



Hone (a J 



* Individual cedents and suggestions (6) 

■ AT 



THANK YOU VERY MUCH FOR YOUR COMMENTS 

Please return the Guidelines and this form" to Mary Seifert at IRA headquarters,- 



APPENDIX „F 

Field Test Reviewers 1 Comments on Question 5: "As a result of using 

these Guidelines, what* changes, if * any, would you make in future equation 

activities in which you might participate? ^ 

I think this wouM hav^ been exceptionally valuable if we had had it before we 

began. It could serve as an outline of things to be sure were considered « 
v even if we excluded them. 

Everything in fchis document should be included in the planning of an evaluation. 

If we had an external evaluation, this would be very valuable in evaluating its 
quality. % 

T would be sure we recorded our plans for followup before We started; and I f d 

record specific plans to implement (or refute) the reports f recommendations. 

This provided a good base to analyze our reading program ^valuation. 

I would demand that our report be more specific and look at 4ore aspects of our 
. "program*' 1 

I failed to realize the many things we have not *been including in our plans t5 
. date. These are far more comprehensive than we were being. I think we 
can do a better job now. 

I'd realize better how limited we've been in the past definitely conduct a 
thorough internal evaluation, since our external once leave much to be 
desired. I guess we sort of " knew that before, but this shows us clearly. 

I would demand a more systematic, thorough evaluation job by the external con- 
tractor now I know what to look for in a good job 

7 ^ . % , 

We would more seriously consider long-tferro followup and a broader look at the 
overall reading program and we'd get more imput from all involved groups. 

More comprehensive planning and Reworking; better dissemination of information^ 
more use in a formative sense tp revise ongoing programs . 

Excellent planning tool I'd use it mostly as we planned the effort so we didn't 
leave out impdrtant elements. 

I'd give more attention to equitable input from those* involved in preparing the 
evaluation, those whose program is being evaluated, and those who will 
implement changes based on the ^evaluation. 

I'd feel comfortable judging th£ quality of an eternal evaluation or»6ne I 
didn't wprk on — if I could use this dpcument. It provides a focus *or 
outline of what to look for. ^ 
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. " , ? APPENDIX G 

- 1 Interview 'Coraments During Field Test-: Recommendations 
* ' Concerning Local Uses^of the Guidelines.' ■ 

I would select portions of the Guideline — not /try to use them all — Choose 
the sections mostj^rlevant to our study before beginning; 
* •» * 

* v rd use all but the technical section (Information Sources) as a guide 
conducting .the evaluation. 

We probably don f t have the* technical -expertise to ensure it, but I'd try to take 
. £S jnuch of this into account ^in our planning as possible. 



I would have our evaluation ^committee screen the set to select the roost appropriate 
questions for us. ' 

I'd select the sections we'd consider to be "hi'gh priority" and Concentrate on 
them rather than -attempting to use it in its entirety. . 

I think these are best used as a laundry list that our evuluation N committee can" 
jise to select a subset of items that fit our needs best. I'd suggest such. - 
a step in the Introduction. 

How could we get some training' on how to use this? If one of our group was 
trained, he or she could help the rest offts. 



I'd mark the key parts and leave the rest as backup not irrelevant, perhaps, 
but less important for our purposes. * 

Without s^omeofie who had a better technical and evaluation background to help us, 
I doubt that we could make optimum use ,of the Guidelines. 

One set of written Guidelines will never provide all the a^wers, but this gives 
us a ^good checklist. 

I'd ask any outside contractor to work through this with us so we could % reach 
agreement on what they'd do for us. * 
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